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ABSTRACT f 

^ ^ A survey cpyered categories of budget, organization, 

Staff resources, activities, and reporting techniques and policies; 
the survey revealed Research and Evaluation (R and E) effaces 
received federal or state funding. Therefore, most retain financial 
independence from programs evaluated. The best placement of R and E 
offices in the school district's organisational chart appeared to be 
in direct line relationship 'to &he superintendent. Evalfcators 
should be 'independent and their off icfe should not be in the same 
branch as curriculum and instruction > the staff of most R and E 
offices -is small and composed ol members having* various educational 
backgrounds and 1 degrees. Two thirds have a relatively high ratio of 
support to resource staff. Many units also hatte access to computer 
facilities.^ R and E activities center -around testing and outcome 
evaluation.' Maintenance of information files and preparation of 

statistical reports were also significant activities oi the staff. 

The most common data collection technique employed seemed to be tests 
and questionnaires, supplemented -by other collection methods. R and E 
staff are sole determiners of" report content or have ultimate 
authority over report contents. Several suggestions are offered for 
\ine*:easincf evaluation use. (DWH) ' 
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Within the past several years, many large school systems have established 
offices of research and evaluation in response to public d emands for account- 
ability ♦ However, the internal organization of R & E offices, their placement 
withixi the overall administrative structure of the system, and the services 
they provide vary considerably from district to districtT Some factors which 
may have contributed to this variability among the offices include (a) varied 
perceptions of their roles and responsibilities, (b) lack of well-defined goals 

and purposes, (c) political struggles for the control of information, (d) limited 

I - 

awareness of existing effective public school evaluation units, and (e) the^re- 
active development of R & E offices in response__to federal and local demands ♦ 

The objective of this study was to examine, via distribution of a question- 
naire, the current status Qf R & E offices in medium and large city public school 

systems and to describe the organizational models used to provide evaluation 

I 

services. Kean (1980) highlights the importance of this type of investigative 

study in resolving conflicting demands made upon interned offices of research 

• i » * 
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and evaluation. The present research" activity was motivated Jay a task assigned 
to administrators in diaries ton County, that of reorganizing the administrative 
levels of the school district. It was intended that a major outcome of the 
study would be information to "assist them in making decisions about (a) the 
appropriate placement of R & E within the system, (b) the relationship of R & E 

to other segments of the system (e.g., curriculum, data processing, pupil ac- 

i 

counting, etc.), (c) the internal organization of R & E, (d) the relationship 
of job assignment to, funding source, (e) the types of activities R & E should 
engage in, and (f) budget allocations. Other areas which are troublesome and 
of particular concern to R & E staff ,were also studied. These included: the 
extent to which districts contract for external evaluations; clerical and 
other support services available within the R & E office and those provided byj 
other district offices reporting techniques and dissemination policies and 



the control over various aspects of reporting; and the types of "research" 
activities in which ;R & E staff engage. 

s Method. 1 

t 1 

The questionnaire . | '* I 

I 

The "Public Schools Research and Evaluation Organizational Survey" is 

• * I ' 1 

an eight-page questionnaire with items covering the following five categories: 
budget; organization; staff resources; activities; and reporting techniques and 
policies. A final question concerned the future of R & E operations. Of the 
approximately 30 items, nearly all were open-ended and required the respondent^ 
to enter specific information or describe situations or perceptions. Although 
the open-ended format was not/conducive tb <easy analysis, the nature of the 
information sought could not be obtained in any other manner. The questionnaire 
also asked respondents , to forward copies of organizational charts. 

i * • * 
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Survey Procedures aiyi Sample 

- The questionnaire As distributed to 200 directors of R & E units in 
Novenber,, 1981. Names of directors were obtained from one of two lists: 
Directors of Research and Evaluation ^from Selected Large City School Districts 
in the United States arfd Canada and the mailing list ojp the^ National Association 
of Ttest Directors. After a second mailing to non-respondents, a tital of 58 

J 

questionnaires were returned. Considering the length and format of the survey, 1 
a low response rate (29%) was expected. j 

The first page of the questionnaire was a cover sheet on which respondents 
were requested to enter their name, job title, and school district. This 
page was removed before analysis of responses in order to maintain anonyinity of 
the school districts and the confidentiality of responses whicii were attitudinal 
or evaluative in nature. i 

District size, ^orty-six respondents reported student enrollment for their 

" \" 

districts. School district size ranged from 17,000 to 210,000 with a median 
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between 40,000 and 50,000. 1 <• 

' ' " \ ' ' ! • ' 

Data analysis . . \ . ^ . 

! For each item on the survey, percentages of respondents selecting" each 

ll ' " 

response alternative or entering 'specific pieces of information were calculated. 
The analyses of closed-format items included the percentage of non-respondents. 
Responses to open-ended items" were' clustered according to similarity of response < 

and percentages were calculated in terms of persons responding to that item. 

V ' . 

Results \ 

• ' . . . * -V ■ . * 

R & E Budgets - _ \ J_ 



Thirty-seven or 80% of the respondents indicated that they received less 
tnii one per cent of the total school district FY82 budget ^or their offices' 
activities. , Pour respondents received 1 - 1.9 .per cent of the local budget 
and two received 2 - 2.9 per cent: Three respondents reported that their bud- 
et does not include any funds from the local school .district. Table 1 shows 

the relationsip between school district size and per cent of local budgets 

■6 

allocated to R 6 B. . 



Most R & E units supplement their budget with monies received from exter- 
nal sources.- Seventy per cent of the respondents receive some federal grant 
monjey as part of their R.S E>«Sudget, 41% receive some state funds, and 20% re- 
ceive money from other sources (e.g., sale of services, contracts, grants). A 
few R & E units '(20%) rely heavily on outside funding, reporting that 50% or 
more of their budget comes from non-local funds; one-third of these units re- 
ported that all their non-local funds were received from the federal government. 

Management of federal and state. funds . . There appears to be some variation 
in themanner in which monies from federal or state grants used for evaluation 
efforts are* handled, but a clear -picture emerges for tworthirds of the respon- 
dents. For most R & E-units, a set percentage of the total grant is 
rarely earmarked automatically for evaluation purposes. Money allocated for 

: ERJC the_e^aluation.J?f_a^unded program is transferred to the R_s. E_bu_d get with 

0 • 



dollars earmarked for specific expenditures (e.g., personnel, materials, computer 
costs). Table 2 contains the results of responses to questionnaire items ad- 
dressing this issue* - ; - | * 

When asked to identify areas of dissatisfaction with the organization of 
r'& E, ^ne respondent stated that evaluation of federally funded programs could 
be carried out more efficiently if R S E were allowed to assign skills to the 
"task. Currently, in this" particular office a person is assigned to carry out 
the 1 "entire evaluation and can work only on tasks related to? the federally funded 

program. ~* | 

Summary The majority of 'the participating R & E offices receive the 
greater part of their budgets from the local school district 9m confirming data 
gathered four years earlier by Lyon et al. (1978) . However, in almost fell cases 
regardless of school district size, the financial committment to evaluation and 
research efforts is. low on the part of school boards, as reflected in the per 
cent of local funds allocated to R & e' offices. At the time of the Lyon study, 
Webster and Stuff lebe?tm (1978) found that the R & E departments of 35 large 
urban school districts received .15% of their districts'* total educational bud- . 
get. .While 49% of our sample had PY82 budgets greater than .15% of their dis- 
trict 1 s budget (implying a tendency toward greater committment) , the purposes 
for which additional monies were allocated are unknown. For example, the "corn- 
mi ttment H might bej toward mandated testing programs. In any case, R & E offices 
depend upon outside funding to supplement their budgets. When federal or state 

monies are received by the district, it is most common for the portion of the 

> i 

grant allocated to evaluation costs to be transferred to the R & E budget with 
the expenditure of funds specified. Thus, it appears that two-thirds of 
the evaluation units retain financial independence from programs evaluated 
within the boundaries* of predetermined areas of expenditure.' 
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Organization j 

The survey w^s designed to ascertain the. organizational placement of R & E ^ 

- offices within school districts, the relationship between R & E offices and 

1 < ~ 

other branches of school districts, arid the degree of satisfaction R & E directors 

felt with their place in the system. Fifty-seven respondents provided usable 
data for this section. . 

' ^ Organizational charts. An analysis of organizational charts and responses 

to survey items reveals that 86% of the responding offices are separated ory 

1 

ganizaticnally from the branches responsible for curriculum (i.e., content 
area coordinators, program managers), and 91% are totally separate from in- 
struction (i.e.,^area superintendents, principals, teachers) . In only one case 
is R & E subsumed by a branch responsible for curricula ^and, in that case, the 
branch is titled *' Instructional Development." Compared to the figure of 62% 
found by Lyon et al. (1978) for** the percentage of R & E Jiits not located in the 
instructional line, there appears to be a trend toward greater independence from 
instructional staff. Survey results also indicated that R & E and Curriculum/ 
Instruction are at the same level organizationally in 64% of the cases pro- 
?' viding usable data. 

To whom does the R & E director report? Eighteen (32%) report directly 
to the superintendent ,^3l (54%) reporj to someone one level below the superin- 
- tendent, and eight (14%) report to someone two levels below the superintendent. 
Lyon and her colleagues also found that most R & E units report to the superin- 

• tendent through indirect channels. 

/ © • * 

Director satisfaction and dissatisfaction were elicited by an open-ended 
item: "In what ways are you satisfied and/or dissatisfied with the internal 
organization of your office and its^ placement within the district administration? " 
r Two issues mentioned f re iuently in response to this item were (a) separation 
I 1 from the branches of the district responsible for implementing programs and 
O (b) proximity to the superintendent within the organizational structure. Ten 

-hKlL- % y 
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respondents reported they were, satisfied 'because of their separation from „ 

program personnel, and one expressed dissatisfaction with an organization that 
placed R'&E jn the same branch of the. district with programs evaluated. Four 
respondents specifically indicated they were satisfied because they report 
directly to the superintendent, and four expressed dissatisfaction because 
they do not. In additi6n ; ei^it respondents who expressed satisfaction with 

/ 

their organizational placement without specifying their 'reasons report directly 
to the" superintendent or a deputy superintendent and are separate from divisions 
of curriculum and instruction. Seven respondents stated that their relationships 
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with other branches of the district were good^but one said that due to separation 
from program personnel, it was necessary to "earn our way* 1 with them. 

Perceptions of R & E. Respondents were asked "How do you believe most dis- 
trict personnel |v±ew the role of your office and evaluation activities?" Nine- 
teen (33%) reported that their offices are viewed as a helpful,, supportive re- 
source, seven (12%) stated they were viewed positively by some district per- 
sonnel and apprehensively or as a nuisance by others, eight (4%) felt that they 



are seen as credible providers of factual information, seven (12%) believed that 



they are perceived primarily as involved with testing ^nd test interpretation, 

three (5%) believe they are seen as threatening, three felt .activities are not 

5 

generally '^understood, and three felt they are seen as a necessary evil oh burden. 

Office titles. Tiie "official" titles of R & E units provide clues to the 
districts* perceptions of the purposes of these' of f ices which only in some cases 
coincide with actual work performed. (See Activities section.) Overall, six 
different descriptors are used in' the majority of office titles. These des- 
criptors and the percentages of R & E units having that descriptor in its title* 
are as^ follows: Evaluation/Program Evaluation/Program Assessment (67%); Research 
(51%); ?fanning (17%^; Testing (16%);, Development (6*); Counseling/Guidance/ 
Psychological Services (5%); and Accountab Lit% (3%}. Descriptors included in 

\ 
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y 1 only one title are: Accreditation* Statistics; Data Processing; Elementary 

/ Curriculum; Student Information Services; Pupil Assessment; and Public Re- 

/ * * - ' ' 

lations/Negotiations. 

Services rendered 6r purchased. Seventy-nine percent (45) of the re- 

spondents receive computer ahd^data processing services from within the 

school district. Although data processing capabilities are critical to evalu- 

ation'and research efforts (playing a vitai role in test scoring ! jflata analysis, 

: . . ■ • — ' 7 - I 

etc. 7, only nine respondents (16%) reported that they are located :in the same 

w 

Sr«iciro"f"the r school district and-have-the same^supe^vis or as the_ office i of 

° data processing. In most cases, computer facilities are located in the bus- 

■1 - ; - ■ - — 

~ iness or information management branch of the district, and* two respondents 
expressed dissatisfaction with the business orientation of data processing 
personnel in their districts. There was also dissatisfaction expressed by v 
one respondent who nqted that a variety of offices in different branches of 
• the district collect overlapping data (e.g. , attendance, maintenance, dxs- 
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ciplinary action, etc.) and~that the collection and analysis of these data \ 
\ were not coordinated. 

Sorae^ services ^are purchased fro^ outside, the district. Thirty per cent # » 
responded that they H £zecpaently n purchase teist scoring services from external 
vendori, and 32% "sometimes" purchase -such services 9 confirming the high ^per- 
centage of testing units with inside scoring support found by Wilkens (1981) . 
Other services with R & E offices purchase outside the district include con- 
sulting (9%~frequfently; 28%-sometimes) ; printing (^1%- frequently; 5%-sometimes) ; 
keypunching (9%- frequently ; 7%-s onetimes ) ; test construction (4%-frequently ; 

1 . V I*"* 

2%-sometimes) ; and test reports (7%-frequently) • - 

Evaluation services. In 28 cases (52%) , district offices purchase evalua- « 
| < tion servcies from agencies outside tlje school district. Xn 16 of these d\s- 

tricts, outside services are purchased by grant recipients who need an impartial, 

'ERIC * 



third-party .valuation. - One respondent said that R * E advises district offices 
to seek an external evaluation 'for one-time evaluation services, and another 
reports that R S E requests an outside evaluation when a particularly sensitive 
issue is being studied. Other • evaluations which had been, contracted out in- 
cluded evaluations of magnet and middle schools , o organization and office 
management'; inservices and a community opinion • survey . 

In most of the districts responding (71%) , the office which answered 
the survey was the only district office providing evaluation or research ser- 
vices. -In the remaining 17. districts, separate evaluation or research studies 
were conducted by offices of federal or statu programs (8), special education 
(2) , research (2) , vocational education (1) , Indian education (1) , gifted 
program (1) , student personnel^, monitoring (1) , placement and special 
projects (1) , systems management (1) , and special services (1) . 

Summary. The most satisfactory placement of R s E within a school dis- 
trict's organizational structure appears to be characterized by (a) separate 

C 

branches responsible for implementation of programs and evaluation of pro- > 

qrams and (b) direct communication between R & E and the superintendent or 
. 1 ' * ■ 

' deputy superintendent. Regarding services received or purchased by R & E, 

R s E relies heavily -on computer facilities owned and operated by the" dis- 
trict's data_processing_unit (usually located in a different administrative 
branch). In addition, R s E purchases" some^teit scoring services from outside 
vendors, and some units purchase a variety of minor services, such as key- ^ 
punching and printing. Although most large scale evaluations are internal 
evaluations, approximately half of the R S E offices occasionally purchase ex- 
ternal evaluation services. In 29% of the districts, offices of federal or 
state programs conduct separate evaluation studies of special prbjects. 

Most directors of R S E units believe their offices are viewed positively 
by users and providers of evaluation data. Yet, 16% commented on negative 
perceptions held of their office, while .12% stated that they were viewed 

10 
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* positively by soma and negatively .by others. / i~ ^ 

Resources rt . • 

Profess ion ad staff. Respondents were asked to provide data on R & E ' r ^ 

staff* Data included the nuafear of staff members, the highest academic 'degree m 
9 held by each, the degree field, 1 and the* funding source for their salaries. 

The jxuatoer of staff members ranged from 1 to 52, wither median of 5 and 
mean of. 8. Thus, the distribution «f professional s€aff is skewed such that 
most R & E units have smaller size staffs, vhile a few districts have very - o 

' large staffs. In fact, four districts had staff sizes ranging, from 2a to 33, 

^ - ^l^iewo^ufflLtst reported a totalr-of-52^rofe5sional st-aff members. * y 

Degrees of 435 professionals described in the returned quesUorrnaiTes 
ranged from Ph. *D. to high school diploma, with professional^degrees in a variety 
; of areas. The- following^datawre- gathered ^thjg^percentage of professionals 

having different type Agrees: 'Ph.D. (28%); Ed.D. (14%) ; Masters (41%); 

• " ' v. 

Bachelors (14%) ; and high school diploma, (3%) . Degree fields were provided for 

• * 
219 professionals and are reported below as percentages: 

• • * 

Percentages Degree Field . I 

24.5 Educatxon 

Educational research; evaluation; tests and measurement . 

■v 10.0 Educational administration , 

1 8.4 Mathematics • m 

7.2 Educational psychology; sdhool psychology 

« 

6.8 j Psychology ' . % f 

6.0 Educational specialties (e.g., chemistry, music) 

5.6 " Counseling; guidance • 

- 1 I 

- 5.2 Reading; English; language arts , 

4 4 Other fields (e.g., social work; economics;, philosophy, theater) 

•• i ( - ' ■ ■ 

3.2 Elementary education 



Percentages Dsqree' ?ield 

2.8 Curriculum and instruction 



i • 



2. 4 Special education ' «£§ 

1.6 * Statistics 
1 # 6 * Businoss m 

0.8* - Child dtvelopmsnt 

• • * • * 

0.8 Caiaaunicaticns > 

She, educational background of R 6 E professional staff varies considerably, 
cutting .across all fields of education and spilling over into psychology and non- 
related areas. Some staff manors are evaluating programs in areas directly ie r 

lated to their degree field. Ye€, only* a small percentage Of professional staff 

» 

matriculated in graduate programs of educational research, evaluation, tests 
and measurement, or statis'tics. \ , 

* Funding source. As of this fiscal year, 30% of tn^ professional .staff re- 
ceived salaries from non-local sources (i.e., federal or state). Below is. a ^ 
table showing the distribute i' of RSE units with varying percentages of staff 

4 * 

funded with federal or state monies : . | 

~ % 'ilS.^SSt'aSS^ ^L.-a**\**Lt. aga^^jii Unit.' 

# • • • tt 




4 



8 



4 " . '*. ' 12 

4 16 



0 



16 



20 36 



8 



$ „ 44 
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4 - 48 

I 

ib ^ 58 

* _ - 64 



Although 43% of J the R & B units have no or.up to 10% of their staffs 

salaried-with .outside. funds , ^sign^ficantj ^rcenta ge m*)±*™™™**™J 0 * 

of their staff supported by federal or state funds. 

Support staff resources. The que stion naire requested a description of the 
support staff resources available to the professional R S E staff. Of the 48 

R S E units responding to this question, 60% had clerical (i.e., secretarial) 

* • i . \ / - - 

help only. Many of these units, however, also had money fidgeted for temporary 

help and/or had technical support availabe to them from other departments (e.g., 
keypunching from data processing). The remaining 40% employed* technical staff 
(evg., programmers, data technicians, research assistants, testing aides) in 
Addition to clerical staff. 

The ratio of support staff to professional staff ranged from .26 to 2.00 
for the 39JR & -E units giving the exact number of employees in their unit. Ap- 
proximately 23% had proportionally more support staff than professional staff; 
44% had a ratio ranging from .75 to 1.00, while '33% had less than a .75 ratio 
(less than three resource .staff to every four professionals) . , 

One area of dissatisfaction expressed jby questionnaire respondents was the^ 
quantity and quality of R & E staff members. Six respondents indicated that they 
-were either understaffed, or that some of their staff members were not appropriately 
trained. One respondents specifically mentioned -a hiring freezo which had resulted 
in a loss of personnel in 'the R fi E office, and one mentioned a need for a larger 
support staff. 

com puter facilities. Nearly all (93%) of the R fi E offices have access to 
computer" facilities. Of the 51 offices using computers, 79%. use the district's 
° mainframe,, and 35% have telecommunications terminlls^wnich "hook up" with the, 
district's mainframe; Only 16% purchase CPU time" from local Universities as 
the only source of computers' or to . supplement their, use of the district's computer. 
Additionally, 10% either own , or lease a minicomputer or a larger computer, while 
10% use a government-owned computer (e.g., county or state government). 'Other 
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equipment owned by 25% of the R & E units includes word prpcessors, mark sense 
card readers, optical scanner?, and high speed printers. 
~ s^mjir-y. The mA j or ity'ofe^R &^E~oSices have ^ relatively sm^ll slaff coti- " 

pos'ed to members having various educational backgrounds and degrees. Fortyj-two 
percent have doctorate degrees; 41% have n^sters degrees. Non-local monies 
provide salaries for cne-third of these professionals. Two-thirds of the R & E 
units hav% a relatively high -ratio of support to resource staff. While 60% have 
clerical help only, 40% employ other support jstaf f , such as programmers and re- 
search assistants. A majorityjof R & E units also have access to computer 
facilities, in most cases owned By the school district; a few also rely on com- 
'puters owned by local universities. 

Activities . 
» * 

The survey listed 18 descriptors of activities performed by R & E units, 

with space provided for additional activities, the percentage of R & E units 

• " » • — 

involved in each of the 18 activities is listed in Table 3. 

R & E units provide a median of 14 different types of services to their 
pohool districts, with more than half of the units involved in 16 of the 18 
activities listed on the questionnaire. Most common to R & E units is adminis- 
tratidn of norm-referenced tests and preparation of test reports. Implementation 
of criterion-referenced testing programs is the responsibility of over 80% of -the— 
R & E units. Responses to an item concerning minimal competency testing programs 
indicate that the| R & F role in testing has expanded to include MCT programs, witji 
84% nf the R & E units involved in various components of these programs, from test 
development to scoring and reporting. 

R & E is also heavily involved in a variety of program evaluation activities. 
Nearly all R & E units evaluating educational programs are at the front end of 
program development via needs assessment 4nd proposal writing. Process and out^o^ 
evaluation are common activities. 

•o < • ' „ 14 . ' 1 
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Due to R & E involvement in massive data collection efforts (e.g., test 
scores, evaluation data) , it is not surprising that 82% of the units spend time 
managing~the storage , retrieval , and, update ~of "data. Ten of "the 55 districts"" 
added two activities to our list: (a) maintaining data files which are not 
, directly related to testing, research or program evaluation activities (e.g., 

» 

* student demographic data, although surely this type of data could be used for 
such purposes) and-, (b) preparing various administrative reports summarizing 
district data (e.g., Office of Civil Rights reports). 

Other activities in which | over half the R units are involved are pro- 
vision of training/inservices (82%) and selection of students for special pro- 
grams '(55%) . Lesser activities were meta evaluation (27%) and personnel evalua- 
tion (24%) . Other* R S E offices (11%) added that they are directly involved 
in the utilization of their evaluation findings through active participation 
in administrative planning and development of system goals and objectives. 

The survey also requested that the respondent enter the percentage of time 
allocated to each of the activities listed. The activities with the highest 
allocated time, in order, were outcome evaluation, evaluation^esign/plahning, 

m norm-referenced- test administration, and research. 

Research., Although most R5-E units checked "research" as an activity con- 
ducted by their staff, answers to a question about research activities gave a 
different picture of the nature of their research activities, as distinct from 
evaluation, and the extent to which research was actually a priority function of 

t * 

R S E. All but three respondents perceived a fairly clear distinction between 
research and evaluation activities. Unfortunately, just over 50% of the units 
stated that they presently do not conduct "real" or "basic" research investi- 
gations which "go beyond the scope of program evaluation." Most of these units 
facilitate the research efforts of outsiders, however. The remaining districts 
emphasize one or both of two types of research activities: (a) research which 



relates closely to program evaluation (e.g., comparison of secondary reading 

t ' ( 

programs? /sustained effects of Title 1 programs; Title I pull-out vs. in-class 

scheduling) and (bTreieirch^slLgnTd to'fieTp shed light on administrative — ^ 

dilemmas, Examples of this latter typ* of research include: the four-day educa- ,\ 

tional work week; effects of multi^grade or combination classes; classroom 

grouping practices; staff absenteeism; factors of effective schooling; student 

absenteeism; the sociometric impact of desegregation; longitudinal studies of 

student achievement; the predictive validity of locally developed screening in- 

struments;. student, parent, and teacher attitude research; class size'. 



Services to outsiders. A final question in this section inquired about 
services provided to outsiders for a fee (such as test scoring or consulting). 
Of the 51 units responding negatively to" this item, three are currently con- 

4 t — 

sidering this type of activity. Four R s.E units offering t*.e.ir„seryice.s_for_a 

fee are doing the following: analyzing proficiency data. for their state depart- 

1 

ment of education;, selling a license to use their tests and scoring test results; 
providing data to external researchers; and receiving reimbursement for evalu- 

ators' time and services. 

Summary. Overall, R S E activities center mostly around testing and out- 
come evaluation. This should not come as a surprise, as Lyon et al. (1978) had 
uncovered .he dominance of student achievement data in the data collection efforts 
of evaluation units. With RtE units responsible for all state- and district- 
mandated NRTj and CRT programs (see Wilkens, 1981), this time-consuming activity 
limits the amount of time spent on other data collection effort's, such as process 
evaluation and research activities. 

Five respondents expressed dissatisfaction' with their districts' emphases 
for R S E activities^ Specifically mentioned was a need for greater emphasis 
on planning,! local research and evaluation, evaluation' of organizational 
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effectiveness, and more research and evaluation (as opposed to evaluation and 
testing)l 

i * -J ' f 

Many offices are involved in other activities, such as training? some ) 

engage in research designed to answer administrative questions and participate 

in administrative decision-mal&ng. The data collection burden placed upon R & E 

units has demanded that R & E staff create and maintain "computerized" student 

* * * »■ 

data files. One outcome of data management is the education of' R & E staff in . 

computer usage and file organization, and their subsequent movement into* the 

rcilm of data processing* As a result, nearly one out of five units returnin g 

■ — — = \ ] ' : * 

the survey, added that maintenance ojf information files and preparation of 
statistical reports were significant activities of their staff. 

9 

Evaluation Models , Techniques and Reports ^ 

_ _ j i ' 

B Q ael — Thirty^seven respondents t68%) reported that they use a model <^r 

variety of models which are generally recognized evaluation models (CIPP: 11; 

Tylerian: 1; Discrepancy: 3; Service Delivery Assessment: 1; Center for the 

Study of Evaluation Model: 1; Hammond-EPIC: 1; locally developed models based 

on known models: 4;. fifteen (27%) , responded that they do not use any particular 

f * 
model, and seven (13%) use ai\ eclectic approach to evaluation design. Ten (18%) 

responded with a statistical analysis technique or research design rather than 

an evaluation model. Such resjponses included Title I models A,_B and C (i.e., 

norm-referenced, experimental, and regression analysis) . 

Evaluation techniques. A wide variety of techniques are employed in R & E 

offices, the most widely used being tests 190%), interviews (64%), survey and/or 

questionnaires (95%) , classroom observation 139%) , checklists (4?%) , and district 

records (9%) . Outcome measures (other than test scores) which are frequently 

used include attendance rates (30%) # dropout rates (13%) , discipline referrals 

(18%) , attitude scales (25%) , self-concept measures (18%) , suspension rates (9%) t 
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time-on-task. (4%), behavior fating scales (2%), mobility* rates (9%), teacher 

absenteeism (4%) , instructional expenditures i (4%) , classroom gradesr (4%) , 

^ai^lisir^ measures (2%); accident rates (2%) , and 

library circulation records (2%) . Eighty-seVen per cent of the respondents 

reported using three or more different evaluation- techniques. { 

Reporting. The most frequently used report formats were oral presentations 

(66%) and written reports~(73%) . Eight respondents said that their written i 

reports generally include modified summary reports or executive summaries. Other 

report formats include , slide and video presentations-, press releases, computer 

• . * . i \ 

printouts', workshops and inservice training, .and test result pamphlets for parents 

A study was also made of the extent to which persons from other district 

jf> * 

offices influence and/or control various aspects of the reporting process. Be- 
spondents-were asked to indicate-who is involved and who has ultimate authority 
over four facets of reporting evaluation Mid research results: report contents, 
format, dissemination schedule and recipients. Thirty reported that only members 
of the R & E staff are involved in determining report contents, while 26 re- 
sponded that both R & E staff members and persons from other offices were involved, 
including program personnel or clietits (16)^and administrators (9). Ultimate 
authority over report contents is held by R & E staff members alone 15T 36 cases 
(64%), but in 15 cases (27%) a superior administrator has final authority over 
report contents. In 38 districts, the same people who are involved in and 
have ult im ate authority over report contents also control format and diss emination . 

Among the eighteen districts where different report activities involve different 

f 

people, the client is more^ involved in decisions involving the dissemination of 
the report, both the schedule for dissemination (5) and the recipients (7). In 
three cases the client has ultimate authority regarding the dissemination schedule, 
and in one case, also controls the specification of recipients. However, in'aix 
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cases, high-level district ^administrators assume sole authority over dissemination 
schedules and specification ofjr.epip.ients,, * 



Use of R & E results. Forty-five (80%) of the respondents reported that [ 
evaluation and research results were used by district personnel to make decisions* 
JDecisionjs cited as examples of use of R & E results included curriculum revision 
(9) $ program modification, initiation, or continuation (21) , provision of inser- 
vices C6), district goals and priorities (11), budget allocations (6 ) f student 

placement and scheduling (8)., instructional ptrategy modifications (6) , personnel 

i * ■* 

policies, placement, or employment ,(6) , improved support and administrative 

services (4) , and building utilization or school consolidation (3) . 

j . 1 
Siimmary. _R.&.E units approach evaluation design from a variety of perspec- 
tives, using recognized evaluation models, statistical techniques, and selected 

°. , * '\ 

designs (their own or others) best fitting a particular program . Tests and 

questionnaires appear to be the most common data collection technique employed, 

supplemented by # other collection methods* 

In just over half the R & E units, R & E staff are the sole determiners of 

s " 

report contents, and in two-thirds, they have ultimate authority over report 
contents, format and/or dissemination* In the 18 districts where clients are 

more influential in some aspects of reporting over "others, report dissemination 

, <» " 

4 

(scheduling and recipients) is the area in which they are most involved* 

i 9 j 
Survey respondents gave se veraL-Bxamples ^^h±ch~-evartuation results were 

used to make programmatic or administrative change^. Although 80% of the re- 

« 

spondents confirmed the use of evaluation results by decision-makers, it was 
not clear wKether use was "immediate and instrumental or gradual and conceptual. - 
(See King et a*, 1981, for an excellent discussion of this distinction.) The 
important point, however, is that results, were perceived as used for decision- 
making. , J* 
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A Final Question 

The last item on the questionnaire inquired about major problems and 



challenges which R s E offices will have to face^in the future. (Results are 

sunwarized in Table 4.) .With demands for educational accountability placed upon 

states and local school districts across the country, coupled with the current - 

state. of financial affairs, RSE departments face a similar, yet very uncertain 

and challenging, future. On one hand, R s E units will be losing much financial 

support along with other district departments. Nearly half the respondents 

expressed concern over the financial future of R & E. Federal funds which provide 

revenue for program evaluation activities in 70% of the RSE units are drying 

I / . 

up at the same time that local school boards are following tTie~natiohwTde trend 

of reducing government spending. On the other hand, R & E is- the most critical 

unit for ensuring accountability at the local level and for providing objective j 

data on which to base administrative decisions. With involvement in large-scale 

testing programs and provision of data for administrative decision-making (e.gl, 

in which areas whould the budget be reduced?) , the demands for RSE services are 

increasing. Yet, fifteen respondents commented on the heavy work loads and in- 

sufficient resources '(e.g., staff computer facilities) needed to maintain 

'<r- J ' " 
existing operations. - 

u . 

, fln additional dilemma which school districts face is the characteristic 
nature of R & E operations. While only a few respondents (3) mentioned a con- 
tinuing need to "sell" RSE services and the utility of its* data for decision- 
making, four other respondents observed that .RSE provides "indirect" services 
and, therefore, is viewed as more expendable than other central office depart- 
ments which are relatively linked more directly to daily classroom instruction. 

' Most respondents observed that the future will necessitate greater opera- 
ticnal efficiency to meet current and future demands' in light of financial trauma 

.20 . 



and to ensure that evaluation data will be used for decision-making. Most often 
cited was a need for management information systems, that is, computerized 
data-bases to store and maintain the wealth of student data such that data are 
easily retrieved and updated. Also needed to meet a future of "doing* more with 
leays" according to five respondents, is a review of current R & E operations, 
with the objective of delineating priorities, policies, and procedures for R & E 
activities so that R & E units can develop better data collection techniques and 
collect usable evaluation data responsive to system needs. \ 

' ^ Discussion 
^Bie rather naive attempt of tl^is investigation into thezworld of school dis- 
trict Research .and Evaluation offices was to find nationwide "evidence" to answer 
"should" questions, such as: where should R & E be placed in the administrative 
structure?; what should be the relationship of R & E to other district offices?; 

what percentage of a schools districts budget should be allocated to R & E services? 

* i 
Questionnaire responses could then be used as ammunition for personal feelings 

# ■* 

concerning answers to -these questions. Instead of finding support, data analysis 
clarified a fetf .dilemmas which. R~v& E offices must soon confront * 

One dilemma concerns the placement of Research and Evaluation offices in 
the school districts organizational chart. The Joint Committee! s Standards 
(19811, as Well as professionals in the fit Ce.g. , Rossi, Freeman, & Wright, 
1979) f agree that evaluators should be as independent as possible from programs 



they are evaluating, and that they should be directly responsible to agency heads. 

1 \ . • r 

!Hiis t arrangemenfc\was the one advocated by our questionnaire respondents. . Thus, ^ 

, * . - \v * ' \ ' . * 

the most appropriate^placement for R & E appears to be location in. a branch dif- ^ 

\ ■ . , 

feren£ from curriculum an^kinstruction, with a direct line relationship to the 
superintendent or Ms/haf^epu* 

There are several advantages ^u*d dissadvantages to this arrangement. One 
advantage is . that ^he ''distance keefc>s evaluators fair and unbiased by limiting 
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the influeftce other agency staff might have on ttosn. Ti^e probability of con- 
flicting^ interests is thereby reduced; misuse of .::;£ oxaati.cn for political 
purposes is minimized; and accurate findings are disseminated to all fight-to- 
know audiences. Such heightened credibility, along with technically sound 
methodology and; effective communication and reporting skills, should encouirage 
use of evaluation data for modifying program components. Yet, the organizational 
distance of R & E from instruction may also increase the potential for non-use, 
as pointed out by King, Thompson, & Pechman (1981). Their review of research 
on evaluation^ use suggests that collaboration and periodic verbal communication , 
between evaluator and decision-maker,, if practical, are likely to improve the 
application of evaluation data by bringing trust and a personal touch to the 
evaluation effort. L forced separation of evaluation from instruction could be 
a disadvantage in that it discourages collaboration, nutures negative perceptions 
and misperceptions of the evaluation service and 3taff (so that evaluators have 
to "earn their-wayJ!- with -clients) , and creates a tendency "for evaluators to be 
less involved in program improvement discussion and action. It is often the 
c^se that evaluators, highly trained in evaluation design and statistical appli- 
cations, are perceived as not having the knowledge or background necessary for 
suggesting specific courses of improvement. Such perceptions can be altered 
"as a result of periodic interaction between evaluator and client.* Standard B7 
states that M eyaluators must' not assume that improvements! will occur automatically 
~ once the evaluation report is completed," and that "evaluators . . . should play 
the role ofghange agent." Thus the dilemma:, how can R fc E staff play this 
,role and stimulate arid guide program improvement while maintaining their organ- 
izational distance from the instructional branch? 

The inevitable financial situation contributes to this dilemma while 

creating others. The majority of R & E' offices, like other district offices, will 

1 

be affected to some degree by federal cutbacks- and similar local action, and 
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approximately one of every five will be ^affected severely. Many questionnaire 
respondents stated that they will be expected to meet existing demands with 
jiext* year's limited resources. An obvious consequence is that financial limita- 
tions will increase the probability that evaluation information will not be used. 
Fewer staff and resources could result in less timely and usable information 

if \ 

(a current concern of R & E personnel) ', and , in general, could reduce the amount 

* 

of time and effort put into actions which^ serve to maximize use. „ - 

Another issue axre^eTby^reduced^ budget allocations concerns the types of 
activities in which R S E professionals engage.. There is indeed a need to deter- 
mine priorities for R & E and to restate the purposes of R & E operations in 
light of the recent financial situation; and it may come to pass that whatever 
strides we have made in the direction of describing instructional processes and 
relating fiem to program outcome (using, evaluation models such as GIPP) will be 
•jeopardized. For example, the testing trend has been to supplement norm- 
referenced testing programs with minimal competency or basic skills testing 
programs, thereby "doubling the work load of R & E units with regard to group 
achievement testing. Of course, mandated policies must be first priority.. With 
fewer resources available, additional mandated testing programs to administer, 
and greater data processing responsibilities, it is likely that evaluation will 
be reduced to simplified levels of evaluation design (i.e.-, outcome studies using 
norm-referenced or criterion-referenced student achievement 2ata) and will yield 
little information for identifying why or how such outcomes were produced. If 
R & E doesn't win the battle of "proving" the value 'of their services for high- 
level decision-making, it is possible that R & E.will be gradually transposed 
into an educational data center, divorced from instruction and curriculum 

evaluation. \ ^ 

Another dilemma which R & E office's presently faces and will continue to 
face puts thera in a Catch-22 situation. With budget trimming, RV E can provide 



an essential service in determining policy changes which are related to the 

financial crunch. Unfortunately, many. R 6 E units report that due to a heavy 

work load, the type of research their school district needs is low priority. 

In addition, working against R & E are three, disabilities. (*e is the occasional 

inability of UG offices to respond to immediate and unanticipated demands. 

~ " " " - • 1 ' • . . 

pftea, "ad hoc- data which effectively respond to a particular concern cannot be 

generated. A second disability already mentioned is budget reduction. And third 
is the apparent indirect connection between R & E and classroom instruction. 
For these reasons, R & E is often perceived as expendable by local school boards. 
Thus, many R & E offices have not been given the opportunity" to display their 
potential, and with fewer resources, this possibility is reduced even further. 
In contrast, other offices have been involved in data collection efforts designed 
to shed light on important controversial issues. Given resources and cooperation, 
all R s E offices can provide the type of data upon which administrative policy 
can be based. But, such efforts to be worthwhile, require money and long-range 
planning • * . . " - — ~— — - — 

Alternative Organisational Arrangements 

The situation looks grim |for the many R & E offices hit hard by financial 
cutbacks, especially those still trying to convince decision-makers of the value 
and potential of their services. -We' can't offer tips for changing attitudes, but 
'recommendations can be made for factors .valuators can control. King et al. 
(1981), for- example, offer several suggestions for increasing evaluation use, such 
as improving the quality of the evaluation study and report. Additionally, or- 
ganizational channels can be improved to address some of the dilemmas described 

„ * 

earlier. * * 

A team approach. In most Rfi| offices, a program to be evaluated is as- 
signed to -a single evaluator located in an administrative branch other than' 
instruction, who then becomes responsible for developing the evaluation design, 
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collecting and analyzin^^a^^ re .^ rt - ' 

be created by_this^r^ce^or^««pl^ oxganizationaUy independent .valuator, 
collecting outcome data may not be perceived as "on the side" of the program and 
program manager. . This increases resistance to the eolation of process data . . 
and active .valuator participation' in program improvement efforts. In the minds . 
of urogram staff ," collection of process data becomes associated with the .collection 
of outcome data and with matte* decisions on the continuation of programs. 
These perceptions may result in resistance to the collection and use of process 
data, another problem which tends to limit the use of formative evaluations 
the extent to 'which program managers feel that evaluators, not well-versed in 
tile "content area!" cannot recommend specific changes lor improving educational 
programs. A uean approach whereby a group is assigned to evaluate a program and ' 
r s e staff members .are assigned to different evaluation tasks according to' 
skills^interest, or personality could alleviate biases against evaluators. 
Survey results revealed that R • B staff members come from a variety of educational 
backgrounds. The staff member with background or experience in the particular 
area addressed by a program can be.assigned ke collection of process data, while \ 
a staff member more familiar with outcome measures can be assigned outcome • . 
evaluation responsibilities. Such teams Wd have- two- advantages. First, .the 
collection of process data by a person who is not associated with collecting 
product data could improve relationships between the process evalUator and his/rier 
client and create perceptions of a supportive evaluator. The process evaluator 
could be assigned the task of interpreting outcome -data in terms of process infor- 
mation without posing a threat, while maintaining 1 the objectivity of an outsider. 
Second, process ^valuator's program relevant knowledge and background would enhance 
the credibility of his or her suggestions for program improvement.- 

Pianning. Only 23% of the R & E offices had "Planning" or "Development" in 
their office titles, while 11% reported involvement in administrative^ 

25 - ' 
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long-^ange planning activities. Perhaps planning (i.e., using evaluation data) 
* * * & \ > 

should be formally stated as a crltical # R & E function. Long-range planning - 

. ^ * \ * ' - \ \ r 

activities would also help keep R 6 E at the front end of administrative research, 

tasks. School boards need to be' aware of the need for long-range instructional 

• _ H — ~ ^ 

•planning so that districts* can anticipate financial loss and prepare for the 

, - ! * , 

* \ $ ' • 

. future. % * 

Educational^data-center^ — ThcHgfrting burden of R & E has grown faster than 
the number of ataff members needed to dtouthe job. Little time is "left to col- 
lect other relevant evaluation dUta, while more time is spent on data processing 
tasks. If testing*, scoring 7 and reporting were delegated to anpther department 
such as an educational data center located in the game branch as R & E.and data 
processing, R & E start would be freed to gather critical input. and process 
/evaluation data and to conduct research to answer administrative questions. 
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Table 1 

Number of R & E Uhits Catcgcrizedjay 
ScH^nDl^telc^ize and Per Cent of Local Budget 
Allocated to R & E (N«58) 



♦ 


Per Gent of Local Buc 
Allocated to R & E OJ 


lget 
Jfice 




School- District 
, Size * * 


No 

. Response 


0% 


Less than 
1% 


1-1.9% 


2-2.9% ! 

j 


Ho 

Response 


4 , 




6 


i 


- 1 


Under " 
20,000 


1 


2 * 








20,100 - 
30,000 


3 


1 


1 




* 

1 


« 

30,100 - 
40,000 


• 




8 


1 




4$,ioo - 

50,000 




1 


8 






50,100 - 
60,000 






2 


■ 


•> 


60,100 - 

70,000, 

1 




t 


2 


1 




1 — - 

70,100 - 
80,000 






4 






80,100 - 
100,000 






4 






100,100 - 
200,000 


1 


1 


2 


1 




Greater than 
200,000 






1 






Total 


• 9 


5. 


38 


4 


2 
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Table 2 

" | Management of Federal and State Funds: 

♦ * "When funds for evaluation are received as part of 

. a federal or state grant ♦ . • 



1 • , : 


Nimfcer and Percentage of Respondents (N-50) 


1 

' ** 

Item — 


Almost 
Always 


Usually 


Softe-. 
tines 


Almost' 
Never 


Response 


A set percentage^ of the 
total grant Is auto- 
matically earmarked 
for evaluation costs. , 


4 

(8%) 


10 

(20%) 


16 

(32%), 

1 


15 

(30%) 


• 5- 
(10%) 


Program managers 
negotiate with R & E ' ° 
staff to purchase 
evaluation services. 


6 

(12%) 


( 

11 

(22%) 


13 

(26%) 


16 

(32%) 


4 

(8%) 


: 1 — ■ 

Evaluation dollars are 
kept within the 
program budget and 
are not transferred 
to the R. 6 E office. 


10 

(20%) 


6 

(12%) 


11 

(22%) 


17 

(34%) 


6 

(12%) 


Once funds are transferred r 
to the R & E budget, they 
may be spent as R & E 
sees fit. 


9 

(18%) 


7 . 
(14%) 


7 

' (14%) 


21 
. (42%) 


6 

(12%) 


Evaluation buugets have 
funds earmarked for specific 
„ expenditures (e.g. , person- 
nel, materials, computer 
charges, etc.) . 


24 

(48%) 


10 

(20%) 


4 

(8%) 


11 

(22%) 


1 

(2%) 



Table -3 

Per Cent of R & E Units involved in 
1 Various Activities (N*55) 





aia filje 



Preparation of norm- re f erenoed tes t reports 
Nona-referenced test administration 
Outcome evaluation -v 
Developmtnt of tests and other instruments 
Needs assessment 
Research 

Evaluation design/planning 
Process evaluation' 

Preparation of criterion-referenced test reports 
Provision of training/inservices 

Data management (i.e., maintexfance of student datta files) 
Criterion-referenced test administration 
Norm-referenced test scoring . 
Proposal writing 

Criterion-referenced tes / jLJS coring 
Selection of students for. special programs 
Mesta evaluation 
Personnel evaluation 

Otlierl: . , - 

Maintenance of information files (e.g., attendance, student demo- 
graphic data); preparation of statistical reports (e.g*r annual 
reports , OCR reports , personnel reports f projection reports) 

Administrative planning; financial planning; development of 
system goals and objectives / 

Tfedinical assistance; delivery of consultant services ; to scteoors 
Supervision of institutional (external) research 
Management (i.e. f Title I comparability; district's MBO system) 
Curriculum development ^ j 
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Table' 4 

Number of Respondents Citing „Ijj~~ 
Major- Problems or Challenges for the Future 

! 



Number of Comments 



ERLC 



24 



7 ' 
6 

'5 

5 

3 

2 
1 
1 



7 
5 
3 



8 

3 



Comments 



Financial Problems: - — 

Shrinking budget and funding; difficulty surviving 
budget cuts 

Cause: 

Loss of federal funds; neeS for local support and 
funding 

Declining enrollment 
Related Comment: 

R ; $ E provides "indirect" services and is more ex- 
pendable than other departments; lacks priority 

Unable to expand services to meet district needs 
R & E Operations: . . I 
Need for data-base management, longitudinal files, 

datastorage and update .~ — — % — " " 

Need for better instruments, data collection tech- 
niques, testing program 

Need to collect more, and better evaluation data for 
decision-making; need to make results usable, to be 
responsive to system needs 

Need goals and priorities, framework and structure 
for services, policies and procedures 

Failure to provide timely reports, feedback; 
improved turn-around "time on test data 

Need to reduce costs by* scoring tests locally 

ied for greater operational efficiency 

feed to develop item pools to test local o b 3 e f *ives 



Ne 
tie 



R & E produces too much results 
Vfork Load and Resources: 

Heavy demands; insufficient funds 

Need to improve computer facilities 

Need for more staff; office Is -understaffed 
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(continued) 



Table 4 (continuad) ** 



\ 

: 



Nunber of Comnents 


Comment 


3 
1 

■ 1 t 


Naed for staff training in data processing 
Salaries cannot compete with industry 
1 Need for qualified minorities 

Staff is expanding - need better physical facilities 



0 

[ 

! 
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